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Abstract 

*** ■ ■ t 

A high correlation between on-^line rate of progress and student^ 
Qthievement on a standardized test was found for a coittpiiter-^assisted 
instruction (CAI) program in initial readings In most eases, GAl 



'^^X'i^ti^S'^''--^ 

■•: -^(^li 

measures of progress were better indlGations of spring test" i^iffotm^: -^J^^-^^-^^m 

■J ■„ r.' i' '^•S^^f.'^T'i^ "i.^ 



than was the pretest given in the fall, Rates of progress in the 
©r strands of the CAI program were highly correlated with each otheri . " ..•:jj.,||^^v;^^ 
but certain strands proved, to be better predictors of spring test scofliS- 
than an overall rate measure. 

Regression models were developed to relate spring test scores to 
amount of time spent in CAIj results from these models were in accoie'd^ 
with data from an earlier experimental study designed to evaluate the 
effectiveness of .the CAI program. Using a stepwise regression, whieh 
included both pretests and rates of progress in the strands, multiple ^ 
correlations were obtained of .79 for the Cooperative Primary Test and 
tSk for the Metropolitan Achievement Test. 
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PREDICTING PBADING ACHIEVEMENT FROM MEASURES AVAILABLE ■ 

DURING COMPUTER-ASSISTED INST^.UOTION ' 

J. 0» CampbeU, E, J. *Llndsay, and IV. i\ Atkinson 

Stanford University ■ •' . 

Stanford, California 9i+3C>5 

The Stanford-eomputer-aBsisted instruction ( CAl) program in initiMy^^Jp^^^^^^^^ 
heading has been under development at Stanford University over a penw " 
'of eight years (Atkinson, I968, 197^; Atkinson, Fletcher, Lindsay, ".. " ' ' 

Campbell, and Tarr, 1973).* A recent experimental' study has shovn thai -. ' ■ _^jy^ 
this method of individualized Instruction produces significant gains i'tt 
-reading over what would be expected from classroom instruction alone " ' vV , ■ 
(f-letcher and Atkinson, 1972).- ^ Here we present the results of a cor^" ; ' ■ ^ ' j '^'/f 
relational study relating on-line measures of progress in CAI to posttfst , ..„.^- 

,.a,6hi.e-vetnent« ... 

Computer-asGlsted instiniction is important in teaching reading 
because it provides effective individualized insti-uction. Our inter- 
pt^etation of the ..literatui'e on teaching children to read is that when 
instruction is not indivldua.li2ed, method variables account for a sffl,all 
proportion of the variance in reading achievement. Much of our work is • ^ - 
4imed at making thv teaching sequence sensitive on a moment-to-moment 
basis to the student'^ unique history of pcrfottnance. 

Improving individualized instruction requires accurate estimates of 
the learner •s state of Knt-'v ledge about various classes of itemed for 
. example, sight words at a .^Iven level of difficulty or specific ^■roulfe - 
of spelling patterns. In addition, the relationship between a student's 
state of knowledge at variratj p-^lnts in the CAI curriculum and his 
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-subsequent performance on a standardized test needs to be specified, ^ 
tPhis report presents models fox predicting a student's achievement <)n a • 
'Standardized test from measures taken during CAI in initial reading. 
aim of this study was not to evaluate the effectiveness of the progrin^^,"' 
J'ather it was to assess the predictive power of component's of the CAl.-.-:.:jj?^>- 
program and to develop procedures to identify student strengths and 
weaknesses. ' . 

JQescription of the Reading Program 

The Stanford CAI program is comprised of the following...seven partS| 

'sailed strands. '■, ■ ;„ 

1. Letter Identification . _ . 

2. Sight-word Recognition . ■ 

3* Spelling Patterns ■ -'"-'y-i'^' 

r" 

h. Phonics ■ 

5. spelling 

6. Word Comprehension . 

7. Sentence Comprehension " ^ 
Each strand h^s been designed to provide practice on' a particular set of " 
reading skills, Tn any session the student may study curriculum item? 
f^pom any or all strands. Tne amount of time spent in each strand is 
selected to maximize the student's progression through the curriculum. 

As shown in Figure 1^ entry. into a strand is detemined by. the student's 
level of achievement in the other strands. The student begins with Igttgp 
identification J when he has mastered a subset of letters used in the 
• Initial words of the yigW.-word strand., he begins that strand, Entry 
into the other strands is controlled in a similar fashion, A detailed 
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Figure 1. Initial entry polntd Into strands* Vertical dotted lines 

represent maximal rate contours which control the student's 
progression in each strand relative to the other strands. 
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description and foiinato used in the program are given in Atkinson et ai* 
(l973)t Here we suijimarise ihe major elemente and present a few examples* 
* In each strand the student t>tudies a curriculiim item in several 
, different instructional formats. The instructional procedure varies f^*,Oto- • 
one exercise to the next^ but in each a curriculum item is presented, a . 
ffesponse is elicited from the student, and feedback is given* For exaftif|e.), 
thfe 'recognition exercise in the spelling patterns strand has the following' 
. format! j , ' . v 

, ' Teletype Display Audio Message 

BIKE LIKE STRIKE Type-^TRBCE 
fKtfee words with similar spelling patterns are presented on the tele-*, 

tsrpev/riter, followed by an audio presentation of one of the words* If • 

.c^ . - 

the student types the correct response, a "4". is printed indicating th&t 
the student was correct* In addition, the audio may give a reinforcing 

. message such as '*great" or ''fantastic*' depending on the student's overall 
level of performance* If the student responds incorrectly or exceeds tM/ 
allotted time, the program prints the correct word and gives audio feed^^ 
back, about the nature of the error. 

An example from the phonics strand illustrates the build-a-vord 
exercise; A spoken cue Is given ('^type *stuff*'*) and part of a word is 
printed (ST-). The student must recognize the correct ending from among 
three printed alternatives (-irFF -OP -EP) * In the word comprehension 
sti*and the student is required to select one of three words which fits a 
given categoi^. The student may, for example, be presented with *'CAPrDY - 

' M CAR*' and a^^ked to type the v/ord that is something to eat* Sentence 
comprehension is handled by a fill^in-the-blank fomat where the ?ttudent 

t . ' \ ■ 



is asked to i^cognize the connect item. An, example is, "TED SWAM TO "Jfi- ^ 
With choices^ "STAB RAl^ RUN." Associated with each exercise 1|. 
performance cirltericri that mu^t be met before the student moves ^^j ^^^^^1 ^ 



ftaxt; exercise • ' / ' • .v.: 



Figure 2 illust rates the procedure in each strand for deciding. 
item the student is. to study and in which exercise format it will he J^!$m:X£^^^^^^ 
-.The process shown, is common to all strands except that r ome 



Strands have additional provisions for review and pretest* The items . 
.::fche student is tc study' are sarLpled jf roni a working pool of items <i^^^'>^ .Jl;; . "^^^ 

f rom the master curriculum file, ani-arl presented in one of the exereise- 
.-formats; The sampling continues until each item in the working pool^htl|-- 

been presented, '/fi-ietj this occurs, a decision is made' to shift the stui^|i- 



to another strar\d, to sign the student off the system depending on ^^^ -''■■'^^-^^^^i 



elapsed time, or tc re^jlace those items from ine woi-ktng pool which tp • -.^m^^ 
student has brought to criterion and continue in the current strand. 

The- equipmfc.nt usea is quite simple from a student's viewpoint J a 
model KSR-33 teletypewriter and an audio headset. The instructional 
program is written in SAIL, which is an expanded form of ALGOL, and mm- Jt; 
on a PDP-10 computer. Since the program is dii-ected to students who 
cannot read, .:ipoken cotrxiunication i?. necessary. Digitized audio is us©f5j 
the voice pattern is sampled and the result stored in digital form that 
can be accessed 'as needed to reconstitute the spoken word or phrase. 
The audio system permits fast and essentially random access to any of ■ 
more than 7,000 items. 

V/hen the c.*-.udent logs in at a terminal, his response histoiy is 
retrieved and the instructional materials are selected for the day. The 




Figure 2. Flow diagram for presentation of curriculum items* 
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student may study in one or all of the strands | when he finishes, the 
^:i$tory record is updated and stored in the computer, 

• • . ■ METHOD .. ■ 

Second grade students in the Stanford CAI Reading Program were 
0hosen,as part of the Compensatory Reading Project carried out by lat^M> '"' 
tlonal Testing Service (ETS). The sample of students used in our- study 
was drawn from these second /rade classes and involved 69 students (k^ 
boys and 27 girls). All students received 15 minutes of CAI per day in 
/addition to their normal classroom instruction. The students are pre* 
. dominant ly from lower income Black families. 
: ^est Instruments - 

As part of the evaluation, personnel hired by ETS administered the 
Word Knowledge «nd Reading sections of the Metropolitan Achievement (MAT-) 
and the Cooperative Primary (COO?) tests in October, I972, and again in 
May, 1973. COOP form 12A and Metropolitan Primary I (form G) were Used 
in the. fall, while COOP 12B and Metropolitan Primary t (fom F) were ueed 
in the spring. Scores from the two MAT subtests were combined to yield 
a total reading score. More difficult forms of these tests are usually- 
used in the spring, but ETS decided, given the sample selected^ to use 
for the spring test parallel forms at the same level as the fall tests. 

In this study we evaluate reading gains 'and develop regression modilg 
to predict achievement on the spring tests as a function of rates of 
progress in the CAI strands and fall test scores. The rate of progress 
for each student in a strand of the CAI program was computed by dividing 
the number of items reaching criterion by the time in minutes spent on 
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the strand. Thus, the rate measure repifesents the average time to reath 
©Mterion on an item in a given strand. These rate measures were obtained 
|(?r all strands, except for the Letter- strand} this strand provides an 
introduction to the CAl program and for second grade students is not 
sensitive to individual difftirencei.. An average rate measure over all 
gt rands was then obtained for each student. Since rates varied as a ^ 
function of the strand, a 2-score was computed for each student on ea^ 
. • strand, thereby placing all rates on a common .scale* The ^-scores were 
\:>then averaged over the six strands to obtain an overall rate measure for 
each student. V— 

• Rates of progress in each strand (rather than items covered) wei^ 
used because students differed in the time they spent in the program aiia. 
6n any given strand. The mean CAI time- per student was 18.9 hours with 
a standard deviation of 3.8 hours. In summary, the variables used afe 
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RESUliTS AND DISCUSSION. ' ^ 

AiMevetnent , 

Table 1 sixminarizes achievement on 'pre- and posttests and presents—^ 
H^centile equivalents in comparison to national norms. As expected, 
the student's scaled scores increased during the year on all tests 
iiaired t-tests were significant at the .001 level for all comparisoni|.fe.: 
■Percentile scores based on national noms also increased for all tests* 
ixamination of Table 1 indicates that the pattern of results is'~§lmila|' 
for boys and girls. The fall tests were quite difficult for these 
sfMents; the apparent gains may reflect only a better fit of the test 
%tS their ability in the spring, and should thus be interpreted cautiGTa'ai^/ 
fli' fferences in Rates 

Table' s presents rate of progress in each strand. The total grotif . 
was divided by sex and also into the top, middle, and bottom thirds base4 
on fall scores on the COOP. In this sample the girls progressed fastef • 
than the boys, and the top group progressed more quickly than the middle 
and bottom groups. Note, however, that there is little if any difference 
between the rites for the middle and bottom groups. This lack of dif- . 
ference in rates was also reflected in the spring test- scores for these 
two groups. Even though the bottom and middle groups differed signifi- 
cantly on fall test scores, their spring scores were„nearly identical 
(138 vs. 137 for the middle and bottom groups on the spring COOP and 
ho vs. 39, respectively, on the MAT Total Heading). This result is 
comparable to other findings (Atkinson, I968) indicating that CAI effect© 
the largest relative gains for students at the low end of the distrlteiitloh 
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Table 1 



Soaled Score, Standard Deviation, and National Percentile 
for the Cooperative Priniaty (COOp) and Metropolitan 
Achievement (MAT) Tests 

BEST COPY AVAIUBIS 



MAT Word 
mi Reading 
r Total 



Total Group 



Fall 



Pgrggntlle Mean 



7 

12 
10 
10 



133.1 
33.9 
32.1 
'5l.Y 



S|) 

8.0 
12. U 

10.9 
10.8 



Percentile 

27 
16 
20 
• Ik 



Bering 
1 



Mean 



lUo.3 6.0 
U7.I 12.1 

l+l+.O -u.i' 

l+l+.l - 114 ■ 



dOOP 

MAT Word 
MAT Tteading 
MAT Total 



Fall- 



Boys 



Percentile 


Mean 


sp 


Percentile 


7 " 


132.9 


8.6 


19 


12 


33.7 


12 ..0 


11+ • 


8. 


30.1+ 


10.5 


18 


8 


30.5 


10.8 


12 



Spring 
1 



Mean 

139.3 
1+5. 6 
1+2.5 
1+2.7 



12 

13. r 

11.7 

12.1 



Girls 



COOP 

MAT Word 
MAT Reading 
MAT Total 



Fall 



percentile 


Mean 


SD 


Pe rcentile 


7 


133.5 


7.3 




12 


3I+.I - 


- 13.1 


22 


18 




XX • X 


23 


12 


33.^ 


10.7 


20 



Spring 
1 



Mean 



n 

6*0 
ll.S 

lliO 
9*8 



'•^National spring percentiles are based on test formg other than those 
used for this sample. Scaled scores are basically equivalent across 
forms and levels of the tests, but spring percentiles should be inte 
preted as approximate* 
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Mean Rates of progress in Strands 



• 






Group 








strand 


Total 


Boys 


Girls - 


Top 
Third 


Middle 
Third 


Bottorn 
Thirk 


Spelling 
SD 


(.33) 


.39 
(.36) 


.1+1 
(.29) 


.68 
(.32) 


.25 

(^25) 


.25 
(.20) 


Sight Words 
S'D 


.71 

(.29) 


.68 

(.30) 


.75 
(.27) 


.89 
(.27) 


.63 
(.27) 


.61 

(■]26) 


patterns 
SD 


.71 

(.32) 


.6? 
(*3M 


.77 

(.28) 


.98 

(.27) 


.58 

(.29) 


.57 

(.21) 


Phonics 
SD 


.79 

(.29) . 


.77 

(.33) 


.83 

• (.23) 


1.00 

(.19) 


.69 
(.29) 


.69 


Word Comp. 
SD 


,69 


.65 

(.i^3) 


.75 
(.35) 


1.01 

(.32) 


.30 
(.31) 


(.36) 


Sentence Comp* 
SD 


.63 
(.32) 


.59 
(.33) 


.70 

(.29) 


.91 

(.26) 


(.21+) 


.50 
(.26) 



''■Rates are in items completed per minute 5 standard' deviations for 
these measures are given in parentheses. 



8b 



1 . 



• i-fU: .v:^A' -A ...Vv.-. ..-.liii.j: ^ ....... . : ^ .... ■ - ■■ - - " ■■■■■ . >.^-.'-. ^ . . . — ..^-.^ ■. — — 



; . ■ • ' / ■ ^ 

Cpri^latlon of Rates and Pretests with Spring AQhlevement 

Table 3 presents correlations of rates with sparing COOP and spring 
MAT total scores. Rates in most of the strands are more highly corr^* 
J^ated with spring achievement than are the pretests.. That is, the 
on-line rate measures are a better predictor of student ability than i& ^ ' / 
a parallel form of the test given in the fall. All of the rates are • ;, ■V-<.: ' 

more highly correlated with achievement on the spring Metropolitan tm^k .. 

■■■ / '■ . ■ ■ ■ ' • ■■ ■■ 

•on the COOP tests. Note that the rates of progress in the spelling m$ ■ ' ; 

sentence comprehension strands are more highly correlated with spring , i 
I " aehievement than is the average. These strands are relatively ,diffif^t| •' ■ 

; while the student encounters them after the others, most students "'■ -7'^' \ 

; . • them well before the spring. ' ^ . . J 

Table h presents intercorrelations among variables. The rates ftfi ■ ,^ 

highly correlated with each- other (.81 to .91), indicating that stud&atg 
I' who move rapidly in one instructional strand (for example, on spelling ■ \ 

' ' patterns) tend to move rapidly in other atrands (for example, sight . . . : ■ j 

words and comprehension) . This undoubtedly reflects both an aspect of ^ 
I . student ability and similarities in the strands of the prdgram. In m©st . j 

I cases rates correlate more highly with spring .than fall aehievement. 

i ' Moreover, correlation with .spring achlevejjient is fairly consistent aefoei \ 

\ ' strands. 

Linear stepwise regressions were used to .develop separate models | 

i for posttest achievement on the COOP and MAT Total Reading tests. Table j 

5 presents the regression models multiple R, and the step at which eaeh | 

variable entered, together with the F to enter (Draper and Smith, 1966), • i 

\ • A low cutoff of F - .01 was used in order to include most of the variablgg | 

' • 9 ■ ' .■ i 
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Table 3 

Correlation of Pretests and Strand Bates with 
Spring COOP and MAT Scores 
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0QOP 


MAT 


spelling 


.69 


.76 


Sight Woi-ds. 


.52 


.5-8 


Patterns 


.6h 


.71 


Phonies 


.61 


.68 


Word Comprehension 


.68 


.ra 


Sentence Comprehension 


.73 


.77 


Average Rate 


.69 


.75 


Fall COOP 


.55 


.55 


Fall MAT Word 


.60 




Fall MAT Reading 


.59 


.52 


Fall MAT Total 


.65 


.61+ 



ICS 



cvj 



CO 

I-- 



00 



o 

H 



on 



ON 



00 



o 



o 



CM 



LPs 



ON 



CO 



ON 
ITN 



H 
ITS 



ON 



b: 



ITS 



ITS v£) 
^ CO 



H 



o 



CO 



CO 
lA 



H 

CO 



8 



to 

0 

o 
to 

H 

1^4 



I 



^ CA ^iD ^ O 

ON On ON UN O 



t-. On ON 



vp CM i/N On On Q 
V£i • OO 00 CO CO OD O 



OD VO H 

lA CO CO 00 On 



H 



1 

I 



0} 

I 

CO 

(M 
O 

o 

•H 
+» 

M 
0 

0) 

d 
H 



00 



CVJ 



oo u^ 

00 00 



• 

H 



H 



CM 



CO 



H { CM O 
CTn" 00 O 
« • « 

H 



H O 

CO o 

H 



(1^ 



o 

Q 



oo ^\ 



o 



4) 



lA VO 



i! 

I 

t 



0) 

PC 

I 



4; 



8 



00 



o 

IS 
en 

a 

On 



a 



4) 



H 
H 



9b 



' Regression Equations for predicting Spring COOP 




COOP 



Step ' Variable Regression F %6 

rtimber entered coefficient Multiple R entef. 



1 . Sentence Comprehension rate IC.69 .73 73*^^ 

2 Fall COOP .16 .76 7.11 

3 Sight Word rate -.6.97 •?? ^'^T' 
h Word Comprehension rate 5.27 .79 3.^© 
5 ^ Spelling rate 3.28 .79 l.il 
6 ^ Phonics rate -3.15 ■ .79 .^1 



intercept in IIU.86. Standard Error of Estimate = 3^83. 



MAT Totair'Reading' 



■i-tep 


Variable 


Regression 




F t© 


&mber 


entered 


coefficient 


Multiple R 


enttt 


1 


Sentence Comprehension rate 


13.00 


.77 


98.1')+ 


2 


Fall MAT^otal 


.09 


.80 


9.ii 


3 


Spelling rate 


IhM . 


.82 




k 


Sight Word rate 


-12.38 


.83 




5 


Word Comprehension rate 


2.83 


.83 .. 




6 


Fall mv Word 


.13 


.83 


«3€ 


7 


Phonics rate 


5.06 


.8U 




a 


Patterns rate 


-3.10 


,8k 


.19 


9 


Fall MAT Reading 


.05 - 




.01 



Intercept is 25.36. Standard Error of Estimate ;= 6.65. 
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The rate measures for spelling patterns and average rate did not entgr 
the regression equation with cutoffs at F .01 and Tolerance Level s .01* 
Low cutoffs were used in order to include' most of the variables for 
comparison. 

Under the cutoffs above, average rate did not enter the regression eeiuatisn* 
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for comparison; even with thi. cutoff, the average rate measure did not 
tnter the regression for eithe.- test. The increase in the multiple R 
as each variable is added ia ar. indication of greater predictive power* 
After the sentence comprehension rate and fall test are used, little is. 
gained by adding further variables. The. multiple correlations obtained 
are relatively high and approach the reliability of the tests themselves* 
The f'.ente nee -comprehension rate by itself accounts for most of the vari- 
ance in the posttest data.; rather than .Looking to the average rate as ^ 
simple measure, sentence comprehension rate by itself can in effect serve 
in place of a reading test. Note also that the sight-word rate carries 
a negative regression coefficient for both the COOP and MAT tests, 
although it is positively correlated with both spring tests. 

At a higher F cutoff, say F = 3,5, the results in Table 5 indicate 
that only sentence-comprehension rate and fall COOP would be included 
in predicting spring COOP. Similarly, only sentence -comprehension rate, 
fall MAT, spelling rate, and sight-word rate would be included in pre- 
dicting MAT. The resulting multiple R's are .76 for the spring COOP 
and .83 for the spring HAT J- these compare favorably with the multiple 
R^s of .79 and ,Qh obtained with the low cutoff. 
Relation of CAI_,Tlme tC)^ Spring Achieyement 

We have also estimated the parameters of a linear equation relating 
spring test scores to time on the CAI program. The equations for the 
COOP and MAT tests ai*e as follows ! * 
COOP I3I1.I8 + .32( hours of CAI) ' . . ^ 

MAT --^ 30.97 + .70(houri3 of CAt) 



10 

Id 



The equations are based on correlational data and should not be inter* 
jpreted as suggesting a cause and effect relationship! however, they are 
in accord with earlier results based on experimental evidenoe (Fletohel? 
and Atkinson, 1972). 

■ ■ CONCLUSION • . ^ 

This study has yielded several useful results. First, the high 
correlation of rates with spring COOP scores indicates that progress in 
the CM program is highly related to a student's reading ability as 
measured by standardised achievement tests. Second, the high inter* 
correlations among CAI rates suggests that the several strands of the 
CAl program may be tapping the same skill, or that skills in one strant 
of the program are highly related to skills in others. However, the 
average rate score was not the best predictor of posttest "scores j after 
the first rate measure went into the regression equation, the partial 
correlation of the average rate was so low that it did not enter the 
model under the cutoffs established. Third, regression models fot 
relating test scores to instructional time yielded slope measures of sli 
and .70 for the COOP and MAT, respectively. These slope parameters 
indicate the gain that can be expected with each hour of CAI, and can bi 
used in formulating optimal policies for allocating instructional time 
among students (e.g., see Atkinson, 1972). They can also be used to 
estimate the amount of time needed for a student or group of students to 
i^aoh a given level of reading performance. Finally, using entering 
achievement scores and rate measures from several strands, we obtained 
multiple B*s of .79 for the COOP and .8^^ for MAT Total Reading. 
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Evidence from prior experimental vork has shown that the CAI reading 
pitogram is effecti^'e (Fletcher and Atkinson, 1972). This study has de^ 
veloped regression models' for predicting posttest achievement from 
ifteasures availajDle during instruction. It is interesting to note that 
the rate measure associated with the sentence -comprehension strand is 
highly correlated with posttest scores (.73 for the COOP and .77 for th'g 
Metropolitan; see Table 3). This single statistic proves to be almost 
.as good a predictor of achievement as the multiple R's, In experimenting 
with one or another version of the CAT program, one could use this meaguf^ 
fey itself as a crude but continuous monitor of the effectiveness of an 
experimental manipulation. To the extent that such a measure Is a valM 
predictor of posttest performance, we can reduce the effort and time 
invoiyed in assessing a particular' experimental manipulation. 
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